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Abstract Body 


Background / Context: 

Time to Act, a 2009 report of the Carnegie Corporation’s Council on Advancing Adolescent 
Literacy, concludes thatU.S. students are ill-prepared for the literacy challenges of 2 1st century 
higher education, employment, and citizenship. The poor perfonnance of U.S. high schoolers in 
international comparisons contrasts sharply with the relatively good perfonnance of U.S. 4 th 
graders. The success of 4th graders, and indeed the recent rise in 4th grade NAEP scores, is 
believed to reflect the success of federal and state policies focused on primary literacy. It seems 
we have learned to teach students to read at the 4th grade level without preparing them for 
subsequent literacy challenges. As a result, a high proportion of middle and high school students 
are struggling. These strugglers are ovenepresented in urban districts, among students living 
below the poverty line, and among ethnic and linguistic minorities. 

Improving reading for understanding in the post-primary grades requires exposing students who 
read at all levels to new instructional elements that focus on higher-order comprehension skills 
(e.g., analysis, synthesis, critique, problem-solving). To target these higher order skills, teachers 
need (1) a better understanding of the component skills required and how they develop, (2) a set 
of digestible instructional activities that, if well executed, build these skills, and (3) opportunities 
to leam the classroom discussion procedures that support student progress. High quality 
discussions are critical to three basic components of reading comprehension: perspective-taking, 
complex reasoning, and academic language skill. 

Word Generation (WG) is a research-based vocabulary program for middle school students 
designed to teach words through language arts, math, science, and social studies classes. The 
program consists of weekly units that introduce 5 high-utility target words through brief passages 
designed to spark active examination and discussion of contemporary issues. WG was designed 
with the understanding that promoting classroom discussion can result in particular kinds of 
academic benefits, such as improved word knowledge, reasoning, and expression. The IES 
funded evaluation of WG (as part of the Reading for Understanding initiative) is a school-level 
experimental study that includes two cohorts of schools randomized to treatment and control 
conditions. The first cohort has been studied for three years and the second cohort for two years; 
we present findings after the end of the 2 nd year of the study and at the end of the 3 ld year of the 
study. 

Purpose / Objective / Research Question / Focus of Study: 

In this paper we present data from the years 2 and 3 of the WG evaluation focusing on: 

1 . basic psychometric and descriptive information, by grade level, for measures of academic 
language and perspective taking as well as standardized assessments of student 
vocabulary and reading comprehension (e.g., Word Generation Academic Vocabulary 
Assessment and Global Integrated Scenario-based Assessments, GISA ); relationships 
among these measures and the standardized assessments; 

2. results from impact analyses focusing on WG intent- to-treat effects on (a) standardized 
assessments of student vocabulary and reading comprehension, and (b) new measures of 
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academic language and perspective taking after two and then three years for the first 
cohort of schools, and after one and then two years for the second cohort of schools; and 
3. variation in the effects of Word Generation by levels of program implementation. 


Setting: 

As part of the IES-funded project Catalyzing Comprehension through Discussion and Debate 
(CCDD), the data for this study were collected as part of the impact evaluation of Word 
Generation. In this paper we focus on two grade groups: 4 th and 5 th graders, and 6 th and 7 th 
graders. Cohort 1 includes seven schools and cohort 2 an additional 18. Together, the two 
cohorts represent four school districts. The districts include two major cities serving ethnically 
diverse, low income students; one small city serving ethnically diverse, primarily low income 
students; and one suburban district serving a primarily white, low to middle income population. 

Population / Participants / Subjects: 

As noted above, the CCDD evaluation sample is socio-demographically diverse. For example, in 
district 1, percentages of students in the participating schools scoring below proficient on the 
2013 state ELA assessment ranged from 55 to 79, between 79 and 91 percent of the student 
population was African American or Hispanic, and between 83 and 92 percent were eligible for 
free or reduced price lunch. In district 4, 38% of the students scored below proficient in ELA. 
Seventeen percent were African American or Hispanic and 44 percent were subsidized lunch 
eligible. 

Intervention / Program / Practice: 

Word Generation (WG) is a tier-one, discussion-based program for middle school students 
designed to build academic literacy and academic practices through language arts, math, science, 
and social studies classes. The program consists of weekly units that introduce 5 high-utility 
target words through brief passages designed to spark active examination and discussion of 
contemporary issues. WG was designed with the understanding that promoting classroom 
discussion can result in particular academic benefits, such as improved word knowledge, 
complex reasoning, and perspective-taking. Each lesson is constructed around a text that 
provides a bit of information about a controversial issue, and a few arguments on either side of 
the issue. Some of the issues are local and student-centered, e.g., Should junk food be banned 
from school cafeterias? Should the school day be lengthened? Others are more national or 
global in scope, e.g., Should the government impose a mandatory year of service after high 
school? Should physician-assisted suicide be legal? The launch text introducing each issue is 
written in journalistic-academic language, and incorporated academic vocabulary words, five of 
which are then targeted for explicit instruction in the course of that week. To support students in 
using the target words, there is a debate on the issue of the week. Finally, drawing on the debate 
as well as the texts as a resource, the students are asked to write a ‘taking a stand’ paragraph at 
the end of the week, also using the target words when possible. 

Research Design: 
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Twenty four schools in total were randomized within the four districts, within pairs of 
schools matched on several characteristics such as enrollment, socio-demographics, and 
standardized test scores. The program was implemented and data were collected on all 
students in grades 4 through 7. Data were collected for three years (2011-2014), 
including a pilot year with schools in 6 schools and two years after scaling up to include a 
total of 24 schools (Year 1 and Year 2) with the final wave of data collection completed 
in June 2014. For this paper we employ data from Year 1 and Year 2. 

Data Collection and Analysis: 

Measures 

All relevant participants completed the following group-administered assessments at all time 
points: 

WG Academic Vocabulary. Multiple-choice test that assesses academic words targeted by the 
WG intervention in grades 4 to 8. The majority of target words are selected from the 
Academic Word List (Coxhead, 2000). Each target word is presented in a neutral sentence 
context with four responses to choose from: (a) a synonym (correct answer), an incorrect 
semantic associate, a phonological associate and a nonrelated word. The test includes 50 
items. Percent correct scores were used for analysis (Hwang, Lawrence, & Snow, in 
preparation). 

Global, Integrated Student Assessments (GISA). The Educational Testing Service was 
awarded a grant from the Department of Education as part of the same Institute of Education 
Sciences’ Reading for Understanding Initiative to develop innovative reading comprehension 
assessments. The GISA is designed to be more engaging for students and more useful for 
teachers than previous assessments. These assessments use scenarios, technology, and reading 
strategies to motivate students, to model skilled reading, and to help disentangle key areas for 
improvement. These computer-based assessments are scenario-driven, providing a plausible 
purpose for reading that guides the assessment activities. Example scenarios include having 
students imagine that they are preparing to lead a class discussion or that they are part of a 
study group. All of the activities focus on strategic reading behaviors that skilled readers use 
every day. GISA batteries cover a range of content areas, including Science, Social Studies, 
and English Language Arts and a variety of text types that students regularly encounter 
(expository texts, fiction, e-mail, web pages, and blogs). Skills assessed by the GISA include: 
reading comprehension, perspective-taking, summary writing, metacognition, identifying 
topical vocabulary and learning vocabulary in context, questioning, identifying main ideas, 
sequencing, organizing key words and information, and paraphrasing. 

Core Academic Language Skills Instrument (CALS-I). Group-administered test that evaluates 
students' core academic language skills in grades 4 to 8. The purpose of this test it to assess 
students' skills in understanding, producing, and reflecting upon language forms that are 
prevalent in academic texts (e.g., logical connectives, nominalizations). Tasks assess a range 
of skills through multiple choice, matching, or short written responses. Two statistically 
equated forms with robust psychometric properties comprise the CALS-I: For Form 1 (for 
grades 4 to 6) and Fonn 2 (for grades 7 and 8). Form 1 was used for this study and included 
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six tasks: Connecting Ideas, Tracking Themes, Organizing Texts, Breaking Words, 
Comprehending Sentences, Identifying Definitions. Prior studies have yielded robust 
reliability (.93 as indexed by coefficient alpha and .90 by split half reliability) and validity 
(.70 as indexed by the zero order correlation with the Gates-MacGinitie Passage 
Comprehension). Using Rasch item response theory analysis, factor scores were 
generated using a vertically equated scale. 


The Perspective Taking Survey (PTS). Developed by Robert Selman and colleagues from the 
CCDD team and was designed to assess students’ abilities to acknowledge, articulate, 
position, and interpret the perspectives of multiple stakeholders in a given text presenting a 
social conflict, and provide solutions that consider and integrate their different positions. The 
PTS includes three scenarios and each scenario is followed by 9 open-ended questions that are 
scored using a coding manual that is currently being piloted. In preliminary analyses, the 
coding system has demonstrated good inter-rater reliability among different coders. 

Findings / Results: 

Preliminary data from the first year (cohort 1) of the WG evaluation indicate that, as expected, 
mean levels of the primary constructs of focus increase significantly over the course of one 
school year, and are significantly higher for older grades than for younger (e.g., 4 th graders). In 
addition, the measures of academic language and perspective taking were correlated with 
standard assessments (e.g., with GISA, ~.5-.6), and the intercorrelations did not vary 
substantially by grade. 

Impact analyses using multilevel models with school pair fixed effects revealed a positive impact 
of WG on the most proximal outcome, students’ WG Vocabulary test scores, for both elementary 
(Grade 4-5) and middle grade (Grade 6-7) cohorts in both Year 1 and Year 2. There were not 
significant impacts on any other outcomes in Year 1. However, in Year 2, there were significant 
impacts on perspective positioning, academic language, and deep reading comprehension for 4 th 
and 5 th grade students and for perspective positioning in 6 th and 7 th grade. 

We also explored the impact of WG by implementation level (high, mid, low) on student 
outcomes for each year and each measure, as compared to control group. In Year 1, students in 
classrooms with the highest level of implementation performed better than their control group 
peers in WG Vocabulary, perspective articulation and perspective positioning in 4 th and 5 th 
grades. Elementary cohort students in classrooms with mid level of implementation also 
perfonned significantly better in WG Vocabulary test than the control group. In the middle grade 
cohort, students in WG classrooms with highest and mid level of implementation showed 
significantly better performance on the WG Vocabulary test than control group students. In both 
cohorts, students in WG classrooms with the lowest level of implementation did not show 
significantly different perfonnance in any of the measures from the students in control 
classrooms. 

In Year 2, elementary cohort students in WG classrooms with the highest level of 
implementation showed significantly higher perfonnance on tests of WG Vocabulary, 
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perspective articulation, perspective positioning, and deep comprehension. In addition, students 
in WG elementary classrooms with mid level implementation did significantly better than control 
group students on WG Vocabulary. The middle grade cohort students in WG classrooms with 
the highest level of implementation also performed better than control group in WG Vocabulary 
and academic language tests. Middle grade WG students in classrooms with mid level 
implementation also showed significantly better performance on the perspective positioning 
measure. 

Conclusions: 

These results provide evidence of the impact of the Word Generation program on student 
outcomes in 4 th through 7 th grade. 
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